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(54) ROBOT AUDIO-VISUAL SYSTEM 

(57)Abstract 

PROBLEM TO BE SOLVED: To provide a robot audio-visual system performing the visual 
and auditory tracing of an object to perform the audio-visual servo of a robot, using both 
visual and auditory senses. 

SOLUTION: An auditory module 20 extracts an auditory event 28 by identifying the sound 
source of a speaker by pitch extraction and the separation and orientation of the sound 
source form an acoustic signal of a microphone. A visual module 30 extracts a visual 
event 39 by the face identification and orientation of the speaker from the image of a 
camera. A motor control module 40 extracts a motor event 49 fr-om the rotating position 
of a drive motor. An association module 60 creates an auditory stream 65 and a visual 
stream 66 from the auditory event, visual event and motor event and creates an 
association stream 67 by associating these streams. An attention module 64 performs 
attention control on the basis of the association stream and performs the audio-visual 
servo of the robot. 
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C(D^'>liCUX. T'r>5^3>$lJS[I"=es;^:x-;l/6 4«. 

©^■ffi^RSoWtf a«:S-:5c^r its $ n/c T y s^x 
^-3 >x > y-Atcj: orf*>s^3>$iRaj^tf?:for. 
ujK-> h©taK:i:i^--3K{Ccfc«3^-^$iJiai^t?;x-;i/4 

0<DiES&-t-:$?4 1 <D*Bt®(7>y^>x>^55rtf?'cCCV C 

^-ir h'7-!5'7 o^r/^Lr^-"•$f$lJ®*ix^-';^4 

ojcei^-r^o cntCcfco. ^-^soiai^>^^-;u4 o 

4 5;&5PWMtiJ{BI^?f^j:->r. ffiSfi^-^ 4 1 ^lellglg 
ib^iir. P^'i:' h i 0«:BTS:&f^CCfS]t:t^<fc^CCJ^Co 

[0 0 5 3 ] tr^-T6 B^J. c<DJ:^ccb-c^;s$n 

/c§>^ h A^1:f->'^'cD^ljS±cc^^f ^fccor^ 

- h 6 8 aR(y^'>^ h y-A^i-- h 6 8 btcj:*3a^*r 
-SX h y--A<Dt^^. J:0»*ffltC«)A <W2>l^)Sfl^t»7j^ 
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';-A6 8 b 1 <t. *fflaccj:o^3n-5^^x h 

[0054] *l^?gilJ5Sfl5filK:<i: ^ASP ?K h 1 0 &i 

<7)<fc^c<:l6f^-r^o 5fe-r. HI 0 (A) tCTnt-Jc^^:, 

5o -eur. s 1 0 (B) ^^^Tcfc-^tc. >'N--T'^#iip 10 
#P/>5a*r-^ h 1 o«:«iac.t:< h i o 

omn^t. h 1 0.«. -7^^ 1 6 ;^>«^i*#flD^ 

PcD#^^i&':>r. ^:i:^i^^-;i/2 0:^^#2i^.:6'f^^{4^ 

r V > X- 3 > * ^ - ju 6 0 cceant" ^ « 
[0 0 5 5 ] cnic<fc*). ryi^x-s^3>^t/^-;u 

6 0«, c<DSS:g'<'-i>h2 8(ca'::Jc^r8g:i;;^h»;- 

A2 9«:*^-r^o CCDt^. mtt^t^^-JUS 0(^# 20 

;b6 46*. C(DK«::^ h U-A2 9^ h y Ur. 
[0 05 6] C<DJ:^tcLT, 010 (C) tC7j^T<fc ^ 

sotfSfjjt^i 5(fCj::hmmmP(Dm(Dmm^m^j^A. 

ilB7=^--:$"^-;^3 8{cJ:0«^L, MiSIS'J^?f3&^i^ 

7 0^/rLrrys^x-t/3>^>?jL.-ji/6 otc<s^-r 

J^fc. ^gE#ttI«P<D®:>&iiIf^-a'-<-X3 8tCl& 
[0 0 5 7] C<Dt^. U7i(y h 1 0(i. eK-f-^V h 40 

';-*A6 5Ccj:omK:li^-'l<€r?f'5:Oc<hcc^O. r 
f'>^a>$iJ»'=ei>^-;i/6 4(S. ^cDr'f'>'>3>*iJ 

P^'^^hlOJi, rv5^x-i^3>>; h t;-A6 5CCJ: 
♦3't-^$fJ®*i^:3.-;L'4 0«:$JJWr'2>C<btcj:0#}ra 
#P*iil»b-C. ta:t^t>^-iU3 OO*^'^ 1 5A^# 
»a«P^ffiMUT}S«U#^J:^ic^c-:»Tl^-5o 50 
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1005 8] ^LX. TVt^x->3>^5;'A'-;U6 0 

K^^>?^-^1'2 0CDW^iSiai5]^5 5(^A;^^^ 

^r. ^^ig^iiiss 5»-e<D^^s^^^>tti£*JJgi 

@5S5 3i<:^x.5c ctic-CctO. Mi£$ifiSilgi^5 3^i. 

Xf-:^5 l/)>6O^^a^W*iI[Hl^5 4^J:0©?^-r 
-SCiiCjcO. D>Ky h 1 0«@#(D|^JS:M^brtB 

[0059 ] ccx. =§^^mc^^^mt. 0m^P 

r^i^jr^o ^m^P(omi)mf'-^^-:^3Sic^:^ 

riX(.^^>m^(tC{t. rv2^x-sx3>^^:^jL-ju6 0 

tc^(3:, xxx^^r^3&^?J tmu^P(tcMLxm^ 

l^?5|SsaiilS85 5 6Ccfc*3 rm^ 
>Ptl£SfM@ES5 3CcJ:0#F^^^^tf:^j:C\ xe- 
:^5 1;0^6 r j: -5 C -eXXX$^. <i: O-eSUMCC^sAO 

[0 0 6 0] ^/c. #»a^PCDjK:05|l7'-^'^-X3 8 

-;l/6 0«. *|-iS^>^^-;l/5 O^SOaiLr. 

T, #jjD^P)&i rxxxr-Tc J t^^<o^m^w^^ 

»'::JC>r. #^s2^lPlBS5 5^Ccl:*3 rxxxj ^iS^O 

:^5 ^J:^c-eXXX$^. <i:^-eg|iMCC:feAO 

T^C^ J <t|%^-r^o c©J:^CCOr. a;}<5^ h 1 0 

»s #Jjn^PcDiSia^tfi^cC-^r. Ell 0 (D) cc^fcfc 

o^c. >'s-7'^^»-^cDAis^^^-r'Si^tcc, laiii't 

>?^-jU3 otcr. ^.ig#J!jn«PcDjS<Diii«<!:«Kr rx 

XXJ ^St'-^^^'^-XS 8^cm^^it^o 
[006 1 ] iS/c. A^P?J<'> h 1 0«. =I>.'^'::.^> 

ffln^i^-^:; hibr. ^1 3:5^o'^i 4^#Mur. bir 

QJc-^Ccitif^T^. it-r. AMP5J<-:/ h 1 0{i. l^tCH^ 
mti:lyi''):t^mLX^^h<OXmj:<. t^^{*ai3{C 

A(Difffi€:?f^S:o/Co tj:^. 014 (A) l^P^l^y h7? 
014 (B) «ts*:x h u-A^cj:^ h^ 5»4-> 

t55^. 014 (C) ^3:^:tX h g^-A^cfc-E) h5''>=^>^ 
SrTT^Orti-So CO^:^»;:i-r«. — A©IS#A, B;«)5 
*^4 0#Fa1tcMorS^C0Ti57^>3>i§:tf^'^« 9512^^ 

^ « . WCC «jy TcDji 0 -2) o 
KJMt 1 : AS^fJ^UTT^v h 1 OCDWl^CCA-S, 
>^3.--;l.3 0:^5A£cCDj|l^^UiLTm:l;>f-^> h^^/S 
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^SOt 2 : A^3&ipjp-.r h 1 0(C)lFtLriSUtei(?>^o K 

B^«^^±5^r. ^UWkkxLWnh. tam^i^:x^jU3 

X h v-A7:;i^^n6cDT\ w^B$r^f^&crve>x^s^ 

X-i/3>*i;^;x-;l.6 0tCj:>3B&(7)ta^Xh »J-A 30 
^^iCB^CDTV-^X-t^s^;^ h 'J-A 

6 7;&5^fig3n€>. 

B$^Jt 9 : AJS*^fS^L3^c:^^6. a;i<^; h 1 0<DtaSfF*g 

TVi/x-^'3>-=ti;?^-ji/6 o?&iAB^<z)5S:i:x h 

AS^^'^K:^ h y-A7&5^iS5n'S>o 

B^SiJt 1 0 : Bfii3&^iS^il:ii)-5)o Km'=ei^:x-;U2 0 3&^ 

>^i;^:x-;i/6 03&^B&<z)rvvx-i^3>^fl?^ot: 40 
Bfi;(DS8^;5;hy-Ai3rm«u. W:^:;?; h y-Ao^j&i 

-eur. h 1 OT^^Tr-Vt-a^^A^tCffiJ 

-A6 7 3&s*R83n-5. 

[0 06 2] CCO^fc^tCUr. ±aiL//cS^:M^:tCC*sl^ 

1. 8$SiJt IROH 6Ccr. SrLl>rv>^x->>3> 
;l/6 4Cc*5t:t^Tf->i^3 >?&i5STL/C^TV>'X-ix3 > 50 
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2. ^S>]t4. t 5(<:r. A&;&^mx;^j:<i^:Sci(c 

<fcO. TVv'X-ixaVX h U-A<Otajt:^ h •J-*A:;&s 

3. B#^]t6, t 1 I T vi^x-s^3>x h i; 
AC7>;*:tc<i^Oiisc^^:i:::^ h u - Aic j: 0 Tr-vi^ 3 
1 3«:7j^-rcfc^«:. a.-K^ M 0 3&^ h-^y+^iS^c^Jtlft 
5 (DiEB:^[6icc J: 0 filSI wi^^a-r ^ c i # s J: ^ 

(^fj^-y 1 5CDt^lfrtCClR*^^K iyji^mmZ Og) 
A. 0-J^R^ryi^:r.^iy3>:^b V-l^ 

[0 0 6 3] COcfcOiCUr. AMPJl^i' h 10«. m 

h U-A6 7^*JSL.-C. c<orv$>x-^'a>x h »; 

T) ccii^'rj:o«cta:i:xh •;-ACc^'5>h^>^>5^:05 

i*Wnfc«^Cc«; ai4 (C) {c^n-rJ:^5c. K^fcx 

h lJ-ACCct^rVt^X-i^3 >y?. h ';-A6 7&Cj:-o 
[0 0 6 4] ^£46\ ai4 (B) CC:tel^r. B^glJt 4Jt 

Of 1 5 (om-vmm:^ h v -Aj&ij^^^n. i^tdmm t e 

h »;--A:i:#M-r-6c<Lccctor. ^i^cDh^ v^ 

>i^«:ffill&ctf<c^C<!:A5-c#^o Ml 4 (C) 

CC*5C^t:. B$^IJt 3/)5 2 3#f^ia^r, 'eb-C3 4#i^ 
fflA>6B*SlJt 1 OCDMiS. iEO< A£c&C/B^<DZ::*CD 
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ai4 (B) <c^-rffi:^x h u-A^#ig-r'6ca:<Ccfc 

[0 06 5] ^/c, :3>^^•^:t>ffl^!^^•:^ h<hbr(DA 
P^^^-;. h 1 OU. ^>*-T'</#fln^^*€:S5cM^8iJCc.i: 

[0 06 6] c<o^^(.cLx. ^^mmmmmi>cj:^A 

CCfg^ODjSSliJ&tTJ^j: ^ C <b ^ S i ^t^c . fHJtxj&^cD 

[0 0 6 7 ] ±i3^L:rcm^mP^^C^l^X. AmUTii y h 

io«. 4DOF (SSjK) *W'r^<J:^tc«fi£5nr 
h I 0(.cm.^7>^/ofcm^(iC'O{,>xmjmLtci^K cn 

[006 8] 
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mm'^t^^-}\^t<omm(cj:'ox. mmBL?ymno:>Myi 

> h ^/c«^:i;-^-<> h<D<J5f(cS':Jc^r. Mmx$>^m 

T V X - V 3 > ^ :x - Jl'*^^:!:^- ^ C i :f)^X ^ 

m J: «3 — JliE?it^jri§^CD h ^ ;^ ^^It^ ^ c i 

x^ ^t^^c. mm^cm&L<omn:^ h y - A:s:c;^ti:i:x 

h y-A;&^#iSLri>r^>. cii6©K:i::^ h y-AS 
cnecDiSMx h y-Aso'm:tx h y-A<Dfsjn 

X. unRoMn(om:^^mmuxvL^y hoae^i^ 

^X7^A;«>^fflft$n>So 
[0ffi<3[)ffim^dCltt?B] 

[m2] mi<DAm^^^-j h(Dmmmx$)^. 
mBULAmx^^h. 

IMA] mi(DAmx^yi<y hCC:tefc:f ^ P;K hfm^v' 
[05] m4i^ci6^i^':fuv^ I (Df^M'^iy 

i^Lx^-rmm6^m^<Dr/u y^mx^^. 

me] ^4CC*>tt-S>:/a2;i7 20m^^>^^-Jl/€rfi£ 

ALx^-rmM,mmf&(oy'xn y^mx^^. 
)i^M:ALxmrm^mmf&(D':^^ y i^mx^^. 
i^LX7fi'rmm^mf&(ozru y ^mx;^^, 
i^:^-ji^m:ALX7nrmm)mm<D:/u y {^mx& 

[mi 0] mAr<DUyr^y hW&Miy:^9'A^C^ii^J^- 

[011] m^ooTr^y hmsi:i:>'Xf'Atcfcit^ 

(A) mM'^i^::^-)i^, (B) i^M'tiy^-)\^<D\^:x- 

r(omm0i^^'tmxt>^o 
im 1 2 ymACDUTT^y hmssM:>'Xf'Atc*jii'^ 
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(A) ■=e-^sijai'tt?A-ji., (B) ryvx-^yH> 

[013] S4<Da;f?^ h«eKi^X7^Atcfet:r-5n> 

S (A) h. (B) 'J-/^^^-hJSL 

O' (C) :<7y^lil«:&7n-r0T*^, 
[ia 1 4 ] SI 3CDt6f^WC*5t:f^ (A) P^-;; 

1^. (B) WM::^ h U-ASO' (C) h 'J-A^ 

1 0 A^o^j^^ h 
1 1 
1 2 
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* 1 3 3135 

1 3 a m^^tt 

1 5 (D^K-:, bum 

16, 16 a. 16b -7^ ^ {OTt^y h^M) 
17 UTfi^ hmMn'^:^y'A 

2 0 

3 0 W,n^iypL-j\. 

10 5 0 ^tts-^et^zx-ji. 

6 4 r 7->t^3 >$fl®-=ei^^-ju 

* 7 0 ^^yh'y-Cf 
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(51) Int. CI.' 

G 0 6 T 7/60 
G 1 0 L 11/04 
13/00 
15/28 
17/00 
15/00 
15/22 
15/20 
21/02 
15/02 

H 0 4 N 7/18 



1 50 



F I 

G 0 6 T 7/60 

H 0 4 N 7/18 

G 1 0 L 3/00 



3/02 
9/00 



(##) 
1 50B 5 L096 
Z 
C 
Q 

5 1 1 
545F 
5 5 1 H 
5 7 1 T 
3 0 1 C 
3 0 1 A 



F^-A(#^) 



3C007 AS34 AS36 
KS08 KS18 
KTU KTIS 
WA03 WB19 

5B057 M05 BA02 
DB02 

5C054 MOl CA04 
CC06 EF06 

50015 AA03 CC13 
KK04 LL06 

5D045 ABU 

5L096 BA05 CA02 



C508 3S03 KS04 
KS20 KS39 KTOl 
LT08 NSOl WA02 
VVC07 WC16 
CA12 CA16 DA06 

CA08 CC05 CD03 
FC12 FF07 HA04 
DD02 EE04 KKOl 



FA69 HA09 



